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CALCULUS OF OPERATORS: 
COVARIANT TRANSFORM AND RELATIVE CONVOLUTIONS 
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^- , Abstract. This paper outlines a covariant theory of operators defined on groups 

J^ . and homogeneous spaces. A systematic use of groups and their representations 

allows to obtain results of algebraic and analytical nature. The consideration is 
systematically illustrated by a representative collection of examples. 
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interest [3[[T5H43.50.54.61J. There are some missing connections between two 
periods and the purpose of this presentation is to bridge the gap. 



1. Calculus of Pseudodifferential Operators 

The theory of pseudo-differential operators (PDO) is an important and pro- 
found area of analysis with numerous applications [12,20.51.55,60]. In the sim- 
plest one-dimensional case, a PDO A = a(x, D) is defined from its Weyl symbol 
a{x, £,)— a function on R 2 — by the identity HZl (2.3)]: 



(1.1) 



[Au](y) = 



a(±(y+x),£,)e : 



2ra(y-x)£, 



u(x) dx df,. 



The alternative Kohn-Nirenberg correspondence between symbols and operators fl7l 
§ 2.2] is provided by a similar formula: 



(1.2) 



[Aknu](u) 



a(y, £,) e 



2m(y-x)£ 



u(x) dx d£,. 



There is a natural demand to generalise PDO for other settings. It is common 
to have several competing approaches for this. We briefly outline two of them. 

1.1. Pontryagin Duality and the Fourier Transform. Historically, the theory of 
PDO grown out of the study of singular integral operators (SIO), which can be 
viewed either as convolutions on the Euclidean group or Fourier multipliers. In 
either case, this prompts a consideration of groups and representation theory. 
For simplicity, we take G = (R, +) — the abelian group of reals with addition. Its 
Pontryagin dual — the collection of all unimodular characters X£,i x ) — e 2 ™^ — 
is again the abelian group G isomorphic to (R, +) [27, § IV.2.1]. The Fourier 
transform J maps a function f from the Schwartz space S(G) to f e S(G) by the 
formula: 



(1.3) 



f(« = 



fMxtMdx = 



f(x)e 



-2m£,x 



dx. 



This map is unitary on L 2 ( G ) . Pontryagin duality ensures that the second dual G 
is canonically isomorphic to G and provides an expression for the inverse Fourier 
transform: 



(1.4) 



f(x) 



f (x) e 



2m£,x 



dx. 



Then, one can interpret the formula ^1.2t as follows: 



(1.5) 



[Au](y) 



Xi(v)a(y,i) 



u(x)xeM dxd£,, 



where the symbol a(x, £,) is a function on G x G. Since Pontryagin duality and the 
respective Fourier transforms are readily available for a locally-compact abelian 
group, this viewpoint generates a related theory of PDO on commutative groups, 
see (Ml Part II]. 

The situation is different for non-commutative groups. The dual object G of a 
non-abelian group G — the collection of all equivalence classes of irreducible uni- 
tary representations — is not a group, in general. One can still define the (operator 
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valued!) Fourier transform by the formula 



f(0 = 



f(g)£,(g)dg, where f 6 L^Cdg), £, e G 

G 



and dg is a left-invariant (Haar) measure on G. The inverse Fourier transform is 
not as simple as in the commutative case. For example, on a compact group it is: 

[T- 1 Ti{x)= Y_ drm(£jTr(£,(x)F(£j), 

[£,]6G 

where Tr denotes the trace of an operator. Thus, for a compact group an analog 
of PDO with a symbol a(x, £,) on G x G can be defined by, ES (10.19)]: 

(1.6) Af (x) = Y. dim( ^ Tr (^ W a(x, y f (£,)) ■ 

KleG 

Similar formulae were used in the context of the Heisenberg group 1 3 1 and other 
nilpotent Lie groups [15). 

1.2. Covariant Transform. A different approach starts from the observation that 
operators of spatial shifts f (t) M> f (t — x) and operators of multiplications by 
exponents f(t) (->• e 27tryt f(t) (i.e. shifts in the frequency space) generate the 
Schrodinger representation of the non-commutative Heisenberg group H [17.21J. 
As C°°-manifold H can be identified with R 3 and the group law is: 

(1.7) (s,x,y)*(s',x',y') = (s + s' + ±lxy'-x'y),x + x',y+y'). 
The Schrodinger representation HI is 

(1.8) p(s,x,u)f(t) = e ni{2s+ y (2t - x)) f (t - x). 

We can integrate this representation with the Fourier transform &(x,y) of a func- 
tion ff(q,p) on M 2 QZl § 2.1]: 



[p(fr)fl(t) 



(1.9) 



&(x,u)p(0,x,u)f(t)dxdu 
6>(x,y) e™y( 2t -^ f (t - x) dx dy 
cr(q I(t + 1)) e 2 ™^ 1 - 1 ' f(r) dq dr. 



Up to different letters, this is exactly PDO 1|1.1J| with the Weyl symbol a. 

It may not be obvious that an introduction of the non-commutative Heisenberg 
group produces any advantage over commutative Pontryagin duality. Probably, 
it explains why this direction, rooted in Weyl's original works and spectacularly 
developed in [21,22|, was not widely adopted (see, however, remarkable excep- 
tions |fejnnii3i;i8.49'i). 

Benefits, which can be challenged within the classical PDOs, become more 
explicit when we move to a general setup. A transition to a non-commutative 
underlining group does not become an issue since non-commutativity is already 
in the scheme. Thus, the construction of non-abelian PDOs is different in a com- 
putational sense rather than conceptually. Moreover, historically PDOs appeared 
as "SIOs varying from point to point" and these roots were preserved in H13II14 |. 
We will recover them in examples with the Dynin group below. 
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Such a development is very straightforward for nilpotent Lie groups, as was 
already hinted in fl3l lT4,221. Thus, the concept of relative convolutions l34l was 
initially developed in the nilpotent setting. However, the approach is also usable 
for non-compact non-commutative non-exponential (e.g. semisimple) Lie groups 
as well. The present paper provides a brief illustration to this claim. 

2. Groups and Representations 

Our construction is based on groups and representation theory It is connected 
to the covariant transform [37-39. 44 J, which consolidates a large collection of 
results linked to wavelets/ coherent states 



2.1. Main Examples. For the sake of brevity, we explicate our approach only by 
the following four examples. However, they do not exhaust all possible applica- 
tions. 

2.1.1. The Heisenberg Group. For simplicity, we use only the smallest one-dimen- 
sional Heisenberg group M consisting of points (s,x,u) G M 3 [17l|22j. The group 
law on H is given by l |1.7) . The Heisenberg group is a non-commutative nilpotent 
Lie group with the centre 

Z = {(s,0,0) el, s el). 

The Lie algebra t) is realised by the following left-(right-)invariant vector fields: 

(2.1) S l < r '=±3 s , X l ( r '=±3 x -iu3 s , Y l,1 -'=±3 y + ix3 s . 

They satisfy to the Heisenberg commutator relations [X, Y] = S and [X, S] = [Y, S] = 0. 

2.1.2. Abstract Heisenberg-Weyl (AHW) Group. Let G be a locally compact abelian 
group. Pontryagin duality tells that the collection G of all unitary characters 
of G is a locally compact group as well. For example l27l § IV.2.1], R — K, 
Z = T, T = Z, where T is the group of unimodular complex numbers. The group 
operations on both G and G are denoted by + and their units are written as 0. 

We form a new group G as the set T x G x G with the group law: 

(zi,gi,xi) * U 2 ,g 2 ,X2) = ( z i z 2X2(gi),gi + g2,xi + X2), 

where Z\ G T, gt G G, Xi G G, i = 1,2. In general, G is a non-commutative locally 
compact group. The unit is (1,0,0) and the inverse of (z,g,x) is (zx(g),— Q,— x)- 
The centre of G consists of elements (z, 0, 0), z G T. The left- and right-invariant 
measures coincide with the product of invariant measures of T, G and G. 

For G = M, the group G is the polarised Heisenberg group [17, § 1.2] in the 
reduced form (171 § 1.3]. Thus, for a general G, we call G the abstract Heisenberg- 
Weyl (AHW) group. Another basic example of the AHW group is f = T x T x Z, 
with the group law: 

(zi,wi,ki) * (z 2 ,w 2 ,k 2 ) = (z 1 z 2 w 1 :2 ,wiw 2 ,ki + k 2 ), 
where zt, w- v G T and ki G Z, i = 1, 2 and the operation on T written as multipli- 
cation of complex numbers. Of course, by the Pontryagin duality G is isomorphic 
to G, in particular, T is isomorphic to Z. Our consideration of G shall be com- 
pared with [61]. 
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2.1.3. The Dynin Group. Consider a Lie algebra spanned by the basis {Z, T, U, V, S, X, Y} 
defined by the following non- vanishing commutators ti"3ll : 

(2.2) [X,Y]=S, [X,U]=Z, [Y,V]=Z, 

(2.3) [S,TI = Z, [X,T] = -iV, [Y,TI = |u. 

Thus, the Lie algebra d is nilpotent step 3. It is generated by its elements X, Y and 
T and their commutators. The multiplication on a group D, obtained from 9 by 
exponentiation, is: 

(2.4) (z,t,u,v,s,x,u) * (z'^^u^v^s',*',^') 

= (z + z' + i(st'-s't) + i(xu'-x'u) + l(uv'-y'v) 

+ ^(yx't-xu't + y , xt'-x'ut'), 

t + t', u + u' + i(ut' - u't), v + v' - j(xt' - x't), 

s + s' + h[xy' — x'u),x + x',u +y'). 

A representation dR of c) appears if we extend the representation of the Lie 
algebra f) by the left-invariant vector fields (|2.1| | with the representation of the 
additional operators U, V, T, Z as operators of multiplication: 

(2.5) dR u = xl, dR v = yl, dR T = si, dR z = I. 

This representation is connected with the algebra generated by convolutions on 
the Heisenberg group and operators of multiplications by functions, see Exam- 
ple |4d(iii) below. The group D was used in papers [13.14], thus we call it the 



Dynin group. It is a special (but, probably, the most important) example of meta- 
Heisenberg group [18]. It is also a subgroup of the group studied in (49). 

2.1.4. The group SU(1,1). The group SU(1,1) gejl § IX.l; |7J § 8.1] consists of 
2x2 matrices with complex entries of the form I = _ J and unit determinant: 

M — | P I = 1 ■ The multiplication is given by matrix multiplication and is not com- 

/e 1 * 
mutative. The maximal compact subgroup K of diagonal matrices I n _ i(j 

is isomorphic to the unit circle. Its presence indicates that the group is not ex- 
ponential, that is the exponent map from the Lie algebra to the group is not a 
bijection. This is group is not compact. 

The group SU(1, 1) acts by Mobius transformations of the unit disk (Exam- 
ple |2.1(iv) below) and is very important in complex analysis, cf. [8 J. Intimate 



connections of other subgroups of SU(1, 1) with the hypercomplex numbers is 
described in 139.4011421, we do not touch this interesting topic in this paper. 

2.2. Induced Representations. The general scheme of induced representations is 
as follows, see also (M Ch. 6;[25j § 13.2;|26l § V.2;|33j § 3.1;E3 Ch. 5]. 

Let G be a locally compact group and let H be its subgroup. Let X = H\G 
be the corresponding right coset space and s : X — > G be a continuous function 
(section) l25l § 13.2] which is a right inverse to the natural projection p : G — > H\G. 
Then, any g G G has a unique decomposition of the form g = h * s[x) where 
x = P(g) G X and heH. We define the map r : G ->• H: 

(2.6) r(g) = g * s(x) _1 , where x = p(g). 
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Note, that X is a right homogeneous space with the G -action defined in terms of 
p and s as follows: 

(2.7) g : x ^ x • g = p(s(x) * g), 

where * is the multiplication on G . 

Example 2.1. (i) For the Heisenberg group H we can consider the subgroup 

Z = {(s,0,0) | s G R}. The corresponding homogeneous space is Z\H = 
{(0,x,y) | (x,y) <E M 2n }. Using the maps p : (s',x',y') 1-4 (x',y') and 
s : (x',y') h-> (0,x',y') we calculate the action: 

(2.8) (s^yjax'/y'Wfr + x'^+y')- 
There is also a subgroup 

(2.9) H x ={(s,0,y) eH | s,y 6 R} 

and the respective homogeneous space is parametrised by the real line. 
Using the maps p : (s',x',y') H- x' and s : x' i-> (0,x',0) we find the 
action of H on H X \H: 

(2.10) (s,x,y) :x' ^x + x'. 

(ii) For an AHW group G, there are also two commutative subgroups: the 
centre Z = {(z, 0, 0) [ zeT) and 

(2.11) H G =«z,0,x)eG I zeT /X eG}. 

The natural maps s and respective actions on the homogeneous spaces 
are similar to the above particular case of H ~ R. 
(iii) For the Dynin group B, consider the commutative subgroup 

(2.12) M = {(z,t,u,v, 0,0,0) e B I (z,t,u,v) e K 4 }. 

The homogeneous space M\B can be identified with H through the map 
s(s,x,y) = (0,0,0,0,5, x,y). The corresponding action is in the essence 
the group law ( |1.7) of H: 

(2.13) (z,t,u,v,s,x,y)-(s',x',y') = (s + s' + -(xy'-x'y),x + x',y+y'). 

(iv) For the group SU(1, 1) and its subgroup K we identify K\SU(1, 1) with 
the open unit disk. Defining maps: 

(2.14) p : ( ? ?) M- i and s : z i-> , 1 (* f) , where |z| < 1 

we deduce the respective action: 

az+(3 , foe (3 

z • g = , where g = „ 

y (3z+a y VI 3 a 

This is a linear-fractional (Mobius) transformation of the unit disk [4(J 
Ch. 10]. 
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For G and H, we respectively denote the right Haar measures dg and dh, the 
corresponding modular functions are Ac and Ah- Then, there is a measure dx 
onX = H\G defined up to a scalar factor by the identities [25. § 9.1(5'-6')]: 

(2.15) dq = - — — - dxdh, where q = hsfx). 

A H (h) 

The measure dx transforms under G action j2.7b by see 11251 § 9.1(7'-8')]: 

tt1 « d(x-g) A H (h(x,g)) 

(2.16) — = -—— -, where s x)g = h(x,g s x • g . 

dx A G (h(x, gj) 

In many cases, e.g. for all nilpotent Lie groups Il26l § 3.3.2], unitary represen- 
tations are induced by characters — one dimensional linear representations — of 
its subgroups. Thus we present here the induction from a character only. Let 
X : H — > C be a unitary character of H. Consider the space of functions on G with 
the property: 

(2.17) F(hg)= X (H)F(g) / 

the space is obviously invariant under right translations. The restriction of the 
right regular representation: R(g) : f(g') h-> f (g'g) to this space is called an induced 
representation in the sense of Mackey 11251 § 13.2; |551 § V.2]. 

Consider the lifting L x : C b (X) — » C b (G) of continuous bounded functions: 

(2.18) F(g) = [<C x fl(g)=x(h)f(p(g)) / f(x) e C b (X). 

The function F(g) has the property l!2.17b . The same expression ([2.181 l defines 
a bijection from L_(X) to certain space Lp(G), which is invariant under right 
translations. A right inverse map — the pulling — 7 : Lp(G) — ¥ L (X) is defined by: 

(2.19) f(x) = 0>F](x) = F(s(x)), F(g) 6 L*(G). 

The norm on L* ( G ) is introduced in such a way that both the lifting and pulling 
are isometries. 

Since Lp ( G) is invariant under the right shifts, lifting and pulling intetwine the 
restriction R|l x (g) or ^he right regular representation R with the representation 
p x (g) = J'oR(g)o£ x . It is the second form of the induced representation. Its 
realisation p x in a space of complex-valued functions on X, cf. E5l § 13.2(7)-(9)] 
is: 

(2.20) [p x (g)fl(x)=x(r(s(x)*g))f(x.g} / 

where geG, xgX, heH and r:G— > W, s:X^G are maps defined above; 
* denotes multiplication on G and x • g denotes the action \2.7\ of G on X from 
the right. 

For the case of an unimodular group G and an unimodular subgroup HcG 
(which is automatic for a nilpotent G), the representation (12.20b is unitary in 
L 2 (X). In the case of a non-unimodular (sub)group, we need an additional factor 



A H (h(x,g)) 
A G (K(x,g)) 



to make p unitary, cf. l[2.16ll and ||25l § 13.2(3)]. 
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Example 2.2. (i) For the centre Z of H, the map r : H — > Z is r(s,x,y) = 

(s,0, 0). The character Xh(s, 0, 0) = e 2™n.s f j_ together with the ac- 
tion d2.8l l produces the unitary Fock-Segal-Bargmann (FSB) representa- 
tion C3 § 1-6; M ■ 

(2.21) [p£(s,x,y)f](x',y') =e 7tm(2s+x 'y- x y' ) f(x' + x / y'+u). 

We identify a point (x,y) of the homogeneous space Z\H with the com- 
plex number z = x + iy . Then, the representation 1)2.211 can be stated in 
the complex form: 

(2.22) [p£(s(z))f](z') = e 7tn(zZ '- £z ' )/2 f(z + z'). 

For the subgroup H x | |2.9| |, the map r(s,x,y) = (s + jxy,0,y). A char- 
acter Xh.(s, 0,y) = e 27tihs an j ^g ac y on | |2,10[| produce the Shrodinger 
representation IfTTl § 1.3;|4T1: 

(2.23) [Pn(S/X»y)fl(x') = e™ n ( 2s+2x 'y + ^f{x' +x). 

Clearly, p a (— s, — x, — y) coincides with the representation J1.8|l . Further- 
more, it is known that they are unitary equivalent to FSB representa- 
tion 1(223} . 
(ii) For an AHW group G, we proceed in a similar fashion. The character 
v(z,0,0) = z k of the centre induce the representation on L(G x G): 

(2.24) [pUz,g,xW9',x') = (zx(9'))*n9 + 9',X + x')- 

For the subgroup Hg l|2.11|l . the map r(z, g,x) = [z, 0,x) and the character 
v(z,0,x) — z k induces the representation on L(G): 

(2.25) [ Pk (z,g,x)fKg') = (zx(g')) k f(g + g')- 

(iii) For the subgroup M l|4.8l l of B, the map 

r(z,t,u,v, s,x,y) — [z+ |st + ^xu+ |yv, t, U+ |yt, v — ~xt, 0, 0, 0). 
The character x(z, t, u, v, 0, 0, 0) = e 2mKz of M induces the representation: 

(2.26) [p h (z,t,u,v,s,x,y)f](s',x',y') 

=e 7tih(2z+2s't + st+|(x'y-xy')t + (2x'+x)u+(2y'+ij)v) 

x f(s + s' + j(x'y -xy'),x + x',y +y')- 
In particular, the action of the subgroup M reduces to multiplication: 

(2.27) [p h (z,t,u,v,0,0,0)f](s',x',y') = e 2h » d( * +t *' +ux '- H ^' ) f(s / / x' / u / ). 

On the other hand, the operator p h (0, 0,0, 0, s,x,y) is the shift (12.13b by 
(s,x,y) on H. The corresponding infinitesimal actions are (|2.11 and J2.5b . 
This representation was used in [13J, see also [18. § 3]. 

(iv) For G = SU(1, 1) and H = K we calculate r ( | ? 

character x of K be x I n it jj I = e ^ , then the induced representa- 
tion acts on L 2 (D) as follows: 

(2-28) [p(9)f]( Z ) = ||^Cff|^U|^f^ Z+P 




((3z+a) 2 \fiz+otJ |3z+a \P z + a 
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Since we are not in the unimodular setting now, we calculate the invari- 



i2 



-2 



ant measure on the unit disk to be (1 — |z| ) dz A dz. The represen- 
tation 02.281 1 is unitary and belongs to the discrete series [46. § IX.3]. In 
contrast to equivalent representations used in complex analysis, our ex- 
pression d2.28|) has clearer composition formula, cf. |8. (**)]. However, 
( 12.28b does not preserves usual analyticity We can use either conformal- 
invariant modification of the Cauchy-Riemann equations [39, § 5.3], or 
introduce an additional peeling map, which intertwines our representa- 
tion with the more common one. 

3. Covariant Transform 

Representation theory is behind many important calculations in analysis, this 
is illustrated in the present section. The group-theoretical foundations of coherent 
states/wavelets are well-known and widely appreciated fTl [48l . However, the 
significance of groups and representations in operator theory [32.35.36.39.43J is 
still awaiting its recognition. 

3.1. Induced Covariant Transform. The following definition is a general tem- 
plate, which admits various specialisations adjusted to particular cases. 

Definition 3.1. [37] Let p be a representation of G in a vector space V. For a 
vector space U and an operator F : V — > U, the covariant transform is the map: 

(3.1) LW F v](g)=F(p(g)v), v e V, g e G, 

to U-valued functions on G. In this context we call F a fiducial operator. 

An important particular case of the above definition is provided by a linear 
functional F <G V*, the covariant transform produces matrix coefficients of the rep- 
resentation [24] Ex. 1.1.2.12]. In the case of a Hilbert space V, such a functional is 
provided by a pairing with a vector f G V, which is known as mother wavelet or 
vacuum state [1,48]. Then, the covariant transform becomes the wavelet transform: 

(3.2) v(g) = [W f v](g) := (p(g)v,f) = <v,p*(g)f) = (v,f g ), 

where f g = p* (g)f are called wavelets. The image WfV is a scalar- valued function. 
The scalar case is very important, however, it does not cover all interesting situa- 
tions, see [43 ] and Example 13.71 We also may require a functional F associated to 
a singular mother wavelet, i.e. a distribution, cf. [35. § 2.3]. 

If the representation p and the operator F are bounded, then the image of Wp 
consists of bounded functions on G . Weak continuity of p suffices for continuity 
of WpV. An important property of Wp is as follows. 

Lemma 3.2. The covariant transforms intertwines the left A(g) : f(g') h> f (g _1 g') and 
right R(g) regular representations of G with the following actions of p: 

(3.3) R(g)W F =W F p(g) and A(g)W F = W^^ for all g e G. 

There is the following simple but useful consequence of the above Lemma. 

Corollary 3.3. [395 Cor. 5.8] Let p be a linear representation of a group G on a space 
V, which has an adjoint representation p* on the dual space V*. Let a mother wavelet 
f G V* satisfy the equation 

a(g)p*(g)fdg=0, 
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for a fixed distribution a(g) and a (not necessarily invariant) measure dg. Then, any 
wavelet transform v — (v, p*(g)f) obeys the following right-invariant condition: 



(3.4) Dv = 0, where D = 



with A being the left regular representation of G. 



a(g)A(g)dg, 

G 



As we will see below, the above distribution a is often a linear combination of 
derivatives of the Dirac's delta functions, therefore the operator D turns to be a 
differential operator. Further examples can be found in [44, Ex. 5.9-11]. 

Often we need only a part of covariant transform. For a Lie group G and its 
subgroup H, we fix a continuous section s : H\G — > G, which is a right inverse to 
the projection p : G — > H\G. 

Definition 3.4. (39l § 5.1] Let F : V — > U intertwine the restriction of p to H with 
a character x of H: F(p(h)v) = x(h)F(v) for all h G H, v G V. Then, the induced 
covariant transform is: 

(3.5) rW F v](x)=F(p(s(x))v), vgV, x e H\G. 

Under our assumptions, the induced covariant transform intertwines p with 
the representation induced from H by the character X- To use the condition 1 13.41 
for the induced covariant transform, we need first apply the lifting £ x ([2.181 to 
Wpv and then the operator D. A collection of such conditions d3.4b can charac- 
terise the image Wf V among all functions on X, see Examples |3.5(i)| and |3.5(iv) 
below. 

In many cases, e.g. for square integrable representations and an admissible 
mother wavelet v G V, the image space of the covariant transform is a reproducing 
kernel Hilbert space [1, Thm. 8.1.3]. That means that for any function v G WfV 
we have the integral reproducing formula: 



(3.6) v(-y) 



v(x) k y (x) dx, 



where the reproducing kernel k v provides the twisted convolution with the nor- 
malised covariant transform Wf (p(s(y) _1 )f) for the mother wavelet f, see Cor. 14.81 
For a function v WfV, the right-hand side of (|3.6b defines its projection to the 
space WfV. 

Example 3.5. (i) For G = H, H = Z and the representation p h 02.231 1 on 

L 2 (M), we have p h (s,0,0) = e 27tihs . Thus any function f G L,(R) is suit- 
able for the induced wavelet transform. Explicitly: 



[W f v](x,u)=(p h (x,u)v,f) = 



e 7tm(2yx ' +xy) v(x' + x) f (x')dx' 



(3.7) 



e 2™fu,x" v ( x // + l x ) f( x » _ I x j dx //_ 



The last expression is known as Fourier-Wigner transform (121 § 9.2; [171 
§1-4]. 
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For the representation (|2.22) l and the functional produced by pairing 
with the Gaussian (\>{z) — e _7th|z ' I 1 the covariant transform W^ is: 



(3.8) 



[W^f]( 



^nfiizz' — zz')/2 



f(z + z')e 



-7th|z'| 2 /2 



dz' A dz' 



f(z") e nKzZ " e -KM\z"\ 2 +\z\ 2 )/2 fan A ^ 



where z" — z + z'. This is Fock-Segal-Bargmann (FSB) transform, it 
presents the FSB reproducing kernel k(z, z") = e 7tHzz " e _7th(|z "' + ' z| )/2 . 
Note, that the second exponent is usually attributed to the weight [4: 

uma § 1.6]. 

The image F 2 (C) of 1 13.81 is an irreducible invariant subspace of L 2 (C) 
with the corresponding orthogonal projection: 



(3.9) 



P F : L 2 (C) -*■ F 2 ( 



provided by Q.8|l . The space F 2 (C) is characterised by the differential 
equation (3 Z — z)f = 0, which follows from Cor. 13. 31 with the distribution 

a(s,x,y) =5£(s,x,y) -16^(5, x,y). 

(ii) For an AHW group G, H being its centre and the representation J2.25K 
any function I on G produces a pairing appropriate for the induced co- 
variant transform, cf. d3.7b : 



(3.10) 



[Wiv](g,x) 



x k (g')f(g + g')Ug')dg', 



where we integrate over the Haar measure on G . 
(iii) For G = B, H = M, the representation p h l|2.26b is induced by a char- 
acter of the centre C. Thus, any functional can be used for the induced 
covariant transform to 



[WfW] (t, u, v, s, x, y ) — 



7rih(2s't+st+i(x'y— xy ')t+(2x'+x)u+(2y '+y )v) 



x w(s + s' + \{x'y — xu'),x + x',u + y') f(s',x',u') ds' dx', dy' 

w(s' + |s + |(x"y -xy"),x" + ix,y" + \y) 

i 

x f(s" - |s - \[x"y -xy"),x" - \x,y" - \y) 
x e 27tih(s " t+x " u+ y" v) ds" dx" dy". 

A similarity with the Fourier-Wigner transform (|3.7b is explicit, 
(iv) For G = SU(1, 1), H = K and the induced representation p l|2.28l l, a 
pairing with the function Iq(z) = 1 — |z| , has the property 



(p(h)v,l )=e 2l *(v,lo>, h = 



e" 1 * 



eK. 
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Thus, lo can be used for the induced covariant transform: 

[W v](w) = 

v(C) dCAdC 



wz + 1 /z + w\ ,. . l2 , dzAdz 
D wz + 1 \wz + \ (l-|z|) 



(3.11) =(1 - \w\ 2 



(1-frv) 2 1-|S| 



2 



where C 



Up to the factor ^ 2 discussed in Example |2.2(iv) this is known as the 



Bergman integral [8\. The image space B 2 (B) of Wo is SU( 1,1) -invariant 
subspace of L 2 (D), which is called Bergman space. The orthogonal projec- 
tion: 

(3.12) P:L 2 (D) ->-B 2 (B), 

presented by the Bergman integral (|3.111 l is called the Bergman projec- 
tion. On B 2 (D) the integral <|3. 1H > acts as a reproducing formula, cf. 
The Bergman space is in the kernel of the differential operator — - 



3 £ . For an expression of this operator in terms of SU(1,1) and Cor. 13.31 
see E3 Ex. 3.7(a);B3 § IX.5]. 

3.2. Berezin Covariant Symbol. An important observation |35J is that, for a rep- 
resentation p of G in a vector space V, we have a representation 

(3.13) p(g 1 ,g 2 ):A^p(g 2 )- 1 Ap(g 1 ), ( gi ,g 2 )eGxG 

of G x G on the space B(V) of bounded linear operators on V. 

Definition 3.6. |39l § 4.3; |43| For a fixed operator F : B(V) -> U the covariant 
symbol A(gi, g 2 ) is the covariant transform defined by the representation p and 
the operator F: 
(3.14) 

A(gi,g 2 ) =F(p(gj,g 2 )A) =F(p(g 2 ) 1 Ap(g i )), where (gi, g 2 ) G G x G. 

We also use the notation A(g) for A(g, g). 

Since the covariant symbol is a special case of the covariant transform, the 
respective variants for the scalar case and induced form are applicable as well. 
The combination of both has the special name. For fixed f G V and I € V*, the 
Berezin covariant symbol A(xi,x 2 ) is the induced covariant transform defined by 
the representation p and the functional F(A) = l(Af): 

(3.15) A(xi,x 2 ) = F(p(s(xi) / s(x 2 ))A) = l(p(p(s(x 2 )- 1 )As(xi))f), 

where X\, x 2 e H\G. Again, we denote A(x) = A(x,x). 

As before, this definition is most useful if f and I are eigenvectors for all trans- 
formations p(H), h e H. An important particular case of the construction is a 
unitary representation in a Hilbert space V and the functional 1 e V* be a pairing 
with f € V [5, § 1.2]: 

(3.16) A(x,u) = (p(s(u))- 1 Ap(s(x))f,f) = (Ap(s(x))f,p(s(u))f) = (Af x ,f y ), 

where f x = p(s(x))f, f y = p(s(y))f. 

Example 3.7. There is a large variety of possibilities (even for a fixed group G) 
provided by a selection of various subgroups H, representations p and fiducial 
functionals F. We will illustrate this for the Heisenberg group. Note that, our list 
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is based on the most popular options and is far from being exhausting. For other 
groups, the number of possibilities is not smaller. 

(i) For the Heisenberg group, to make a structure of the listed options we 
introduce a subdivision. 

(a) For G = H and the representation J2.23L take f(y) = l(y) = 5(y) — 
the Dirac delta function. For the subgroup H x | |2.9l l and the homoge- 
neous space K = H X \H representation (|2.23l l acts on [p (s(0, — x, 0)6] (x') 
^x(x') = 5(x' — x). Consider a smoothing operator A : S' — > S, where 

S is the Schwartz space of smooth rapidly decreasing functions on 
the real line and S' is its dual — the space of tempered distributions. 
Then the Berezin covariant symbol is: 

(3.17) A(x 1 ,x 2 ) = (A5 Xl ,5 X2 ), 

which will be related to the Schwartz kernel below. 
The reader may notice that our usage of the Heisenberg group looks 
excessive in this case: shifts on the real line are completely sufficient. 
Thus, we are moving to the next case. 

(b) Again consider G = H this time with the subgroup H x Il2.9b and the 
analogous subgroup H y = {(s, 0, x) 6 H}. Accordingly, for the rep- 
resentation (|2.23b we take f (x) = 1 and l(x) = 5(x) both being tem- 
pered distributions from S'. Then, [p h (0,0,y)f](x') = e nih2x ' v f{x') 
and [p h (0, — x,0)6](x') = 6(x' — x). Since, the Fourier transform of 
p h (0,0,y)f is the delta function 6 y , for the PDO A KN CC3 with a 
smooth symbol a, the Berezin symbol 

AKN(y,x) = a(x,y), 

is its Kohn-Nirenberg symbol a. 

(c) For G = H, H = Z, the representation l|2.22|l and the both I and f be 
the Gaussian (J)(z) = e~ nh ^ z ^ 2 , the transformation (|3.161 l is the Wick 
(or Berezin) symbol of an operator A (HHEl § 2 - 7 '> 22 \- Tne simplest 
calculation of the covariant symbol can be performed for the Toeplitz 
operator l a — PpaPp, with a e L^C) and Pp l|3.9|l . For the Gaussian 
4> and c[) z = p^(s(z))4) we found: 

T a (w,z) = (T Q (f) w ,4) z ) = (P F a4> w ,4> z ) = (acj) w ,PpCJ) 2 ) = (acj> w ,cj) 2 ) 

a(z') e - 7tltl (r^^' + \ w \ Z /'2.+\z'\ 2 /2) e -nn{zZ'+\z\ 2 /2+\z'\ Z /2) fal /\ Qgl 



(3.18) 



_ e -7th(|w| z /2 



+ |2|V2) 



a z 



-nh.{wz.'+zz' + \z'\ ) 



dz' A dz'. 



Clearly, T a (z, z) is not much different from the FSB transform 
of a. 

It is worth to notice, that the unitary equivalent model on the real 
line appears if both f and I are the Gaussians e _7tx ^ 2 on the real 
line. The respective contravariant symbol translates to the language 
of quantum mechanics as the transition amplitude of a quantum 
mechanical observable (in the Schrodinger model) between states 
with minimal uncertainty. 
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Another class of operators with a useful Berezin calculus are compo- 
sition operators [8], i.e. an operator C4, : f 1 — >■ f o 4> for a fixed map 
4> : X — > X of the domain to itself, 
(d) There is another approach for G = H and the representation (12.23b . 
We take an (operator-valued) fiducial operator F : B(L 2 (R)) -> B s (L 2 (R)), 
where B S (L 2 (R)) is the space of bounded shift-invariant operators 
on L 2 (R). F is defined by: 

(3.19) F:Ah>A , such that lim|||M 6 AM 6 - A ||| = 0, 

5— >0 

where M§ is an operator of multiplication by the indicator function 
of 6-neighbourhood of the origin and ||| • ||| denotes the essential 
norm (modulo compact operators). The limit exists for operators of 
local type l52l . 

In particular, for the operator M f of multiplication by a function f (x) 
we have FMf = f (0)1. Therefore, for the representation p h d2.23|) , 
we obtain the eigenf unction property Fp n (s, 0,y) = e 27tlhs I for all 
(s, 0,y) G H x d2.9b . Thus, we can use the induced form of the co- 
variant symbol | |3.14|| only for values p h (0, x, 0), where x G R — 
H X \H | |2.23[I — they are shifts on the real line. Thus the localisation 
map d3.19b defines the covariant transform A x = F(p h (0, x, 0)Ap n (0, — x, 0))- 
the local representative of the operator A at a point x [43 , 52 1 . 
Another important example of operators of local type are SIOs — 
convolutions on R with singular kernels — moreover, F(S) = S for 
any SIO S with a homogeneous kernel. This recovers Simonenko's 
localisation technique for the calculus of operators generated by 
SIOs and operators of multiplications [43, 52, 53 J. 
(ii) For an AHW group G, we can essentially repeat all approaches for H 
described above. For example, we provide an analogue of l3./ffi)(b) 



For 

G generated by a commutative group G consider subgroups Hg l|2.11t 
and the similar subgroup H G = {(z, g, 0)}, the respective homogeneous 
spaces are G = H G \G and G = Hg\G. Take f(g) = 1 on G and 1(g) = 
6(g). For the representation l|2.25b , we have [p 1 (l,0,x)fl(x') — x( x ') an d 
[p 1 (l, — x, l)6](x') = 6 x (x') = 6(x' — x). Then, the Berezin symbol of the 
operator A l|1.5|l is: 

A(x, X ) = (A X ,6 x ) = a(x,x), 



i.e. the symbol a entering the integral d2.25b . Other variations of I3.7fri)(a) - 

|(i)(d)| can be obtained in similar ways. 
(iii) For G = B, H = M and the representation | |2.26|| , we can use the localisa- 



tion approach from [3]/ [i)(d) For a localisation functional F at the origin 
of H similar to d37l9|> , we calculate F(p h (z,t,u,v, 0,0,0)) = e 2 ™ hz I for the 
representation d2.26b and (z, t,u,v, 0, 0, 0) in the subgroup M l|4.8|l . Thus, 
it is sufficient to perform the covariant transform l|3.14t for [p h (0, 0, 0, 0, s, x, y ) 
which are shifts on H. In this way we recovered the calculus of SIO on 
the Heisenberg group initiated in |13 14J, see also [28. 29. 31 1. This can 
be extended to more general nilpotent Lie groups. To this end we need 
to consider a suitable group of dilations, which acts by automorphisms 
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of the nilpotent group fll6l § l.A]. Such a covariant calculus was recently 
considered in (43l . 
(iv) For G = SU(1, 1), H = K and the representation Il2.28b we can follow the 
suit of l33{i)(c]1 by setting f(z) = l(z) = 1 — |z| . The Berezin symbol J3.16l > 



is well-known [6] and very important in the theory of operators |8: 47, 
§ B.4.1.8; 59. § A. 3]. Similarly to the Heisenberg group, the simplest 
calculation of the covariant symbol appear for the Toeplitz operator T Q = 
PaP, where a(z) e L^B) ant ^ P * s me Bergman projection ||3.12| |. Using 
expressions from Examples |2. 1 (iv)] and |2.2(iv) for the lo(C) = 1 — \z\ we 




(3.20) 

Another opportunity to investigate operators on the Bergman space is 
the localisation technique similar to !3./ffi)(d) The localisation can be 
combined with the Berezin calculus (59). 

3.3. Calculus of Covariant Symbols. If a functional F and a representation p are 
both linear, then the resulting covariant transform Wp l l3.ll is a linear map. If Wf 
is injective, e.g. due to irreducibility of p, then Wp transports a norm ||-|| existing 
on V to a norm ||-|| F on the image space WpV by the simple rule [44J: 

(3.21) ||u.|| F '■— IMI / where the unique ve Vis defined by u = Wpv. 

By the very definition, Wf is an isometry (V, ||-||) — > (WfV, ||-|| F ). Moreover, if the 
representation p acts on (V, ||-||) by isometries then ||-|| F is right invariant due to 

Lem.|321 

In most cases, the transported norm can be naturally expressed in the original 
terms for G. For example, for a square integrable modulo a subgroup H rep- 
resentation p and an admissible mother wavelet f <G V the transported by 1 13.21 
norm coincides with the L 2 -norm on X = H\G. Explicitly, for vn £ V and 
vi,2W = (v u ,p(sW)f) v (TJCh.8]: 



(3.22) (vi,v 2 ) v = <Vi,v 2 ) w , where (vi,v 2 ) 



Vi(x)v2(x) dx. 

X 



Another example of a transported norm is the norm on the Hardy space in the 
half -plane [44J. 

The particular case of the above transportation is provided by the Berezin 
transform. For an operator A on a normed space V, the norm of A has the 
standard definition: ||A|| — sup,,,,^ ||Av||. For an isometric representation p of 

G on V and ||f|| ^ 1 and ||l|| $J 1, the associated Berezin transform A(x,u) (|3.161 
is a function on X x X bounded by ||A||. The opposite statement — boundedness 



lb 



VLADIMIR V. KISIL 



of A(x,y) implies boundedness of A — is a variation of the reproducing kernel the- 
sis (RKT) [47, § B.4.1.8]. Another related topic is a connection of compactness of 
A and vanishing of A(x,y) "near to the boundary" [8,9]. We return to RKT in 
Subsection 14.31 

The isometric property 03.22b allows us to follow § 1.2] and deduce compo- 
sition rule for Berezin covariant symbols t35l Prop. 3.2]: 



AB(x,y) = (ABf x/ f y ) v = (Bf x ,A*fy) A 



(3.23) 



Bf x (z)A*f y (z)dz: 



(Bf x/ f z )(Af z/ f y )dz: 



(Bf X; f 2 )(f 2 ,A%)dz 

B(x,z) A(z,y) dz. 



One can note, that covariant symbols behaves (up to the order of A and B in the 
last integral) like integral kernels and this is not a simple coincidence, see below. 
Our formula is more straightforward than the original (5] § 1.2] since we do not 
need a normalization. 



Example 3.8. (i) For the Heisenberg group in the setup of |3./f[i)(a)} the fidu- 

cial functional of pairing with f = 5 produces the identity operator in 
v(x) = (v, f x ) Il3.2b . Since we have the isometry (13.22b in the trivial way, 
the composition rule (13.231 1 follows. Keeping in mind that the Berezin 
covariant transform ||3.17| | is the Schwartz kernel with reversed argu- 
ments, we obtained the well-known integral formula for the composition 
of Schwartz kernels. 



In the setup of [37/ [i)(c) the covariant transform turns to be a repro- 
ducing formula 03.8I > on the FSB space, thus, is an isometry. The speciali- 
sation of the composition rule 03.23b for the Toeplitz operators in the FSB 
space can be found in many works starting from [4J. 

(ii) For an AHW group G, we can essentially repeat all approaches (Schwartz 
kernel, PDO-type, Toeplitz operators and localisation techniques) which 
are in use for the Heisenberg group with respective norms and composi- 
tions formulae. 

(iii) For the Dynin group D and the representation 02.26b , we recall the lo- 
calisation context from !3./|ri)(d) and 3.7(iii) Let Po,5 be the projection 
provided by multiplication with the characteristic function of 6-neigh- 
bourhood of G H. Then, the representation J2.26b produces similar 
projections P 9/ 5 for an arbitrary g G H. For an operator A on L 2 (H) 



we can build a Berezin covariant symbol Ag(gi,g2) = P g2 ,5AP s 



1,6- 



If 



A is an operator of local type [52|, then A§(gi,g2) is a compact for all 
6 < |gi — g2|. Thus, modulo compact operators the symbol A(gi,g2) — 
lim§^o As(gi,g2) vanishes outside of the diagonal. Therefore, the co- 
variant symbol becomes a field of local representatives A(g) = A(g, g), 
g G M 11311251 . The isometry 03l!l) becomes |||A||| = sup g ||A(g)||. The 
composition rule 03.23b reduces to point-wise multiplication of local rep- 
resentatives: AB(g) = A(g)B(g). 
(iv) For G = SU(1, 1), H = K and the representation 02.28b we also have 
the reproducing formula 03.11b on the Bergman space. The respective 
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composition formula for Toeplitz operators is well-known § 4.2; |8) 

m § a.3]. 



4. Relative Convolutions 

4.1. Integrated Representations and Contravariant Symbols. Let G be a locally 
compact group, a left-invariant (Haar) measure on G is denoted by dg. Let p be 
a representation of the group G in a vector space V. The representation can be 
extended to a function k on G though integration 



(4.1) p(k) 



k(g)p(g)dg. 



G 



In the simplest case k has scalar values, however, the same formula is meaningful 
for functions with values in operators on the representation space V. 

The integral d4.1b can be defined in a weak sense for various combinations 
of functions and representations. One of the natural setups is a bounded (e.g. 
unitary) representation p and a summable function k. In this case we obtain a 
homomorphism of the convolution algebra L X (G, dg) to an algebra of bounded 
operators on V: 



p(ki)p(k2) = p(ki *k2), where fci*k2](g) 



G 



ki(gi)k 2 (g 1 1 g)dgi. 



For a representation p induced from a subgroup H, all operators p(H), h. € H 
act in (|2.20b locally. That becomes especially trivial if p(h) are scalars. Thus, for 
induced representations, we are mainly interested in the "complement" H\G in 
the expression 1)4. lb . For a continuous section s : H\G — > G, we rewrite J4.1|l to 
become an operator of a relative convolution 



(4.2) p(lc) 



k(x)p(s(x))dx, 
x 



with a kernel k defined onX = H\G with a (quasi-)invariant measure dx 02.151 1. 
Again, the most natural domain of this definition is a bounded representation p 
and a summable k from Lj(X, dx). Furthermore, in many cases we need to (and 
can) extend meaning of | |4.2|| for suitable functions and distributions, e.g. the 
Dirac delta function and its derivatives. 

Example 4.1. (i) We already mentioned that relative convolutions gener- 

ated by the Schrodinger representation ( 11.81 1 of the Heisenberg group 
are PDO G2D- In this case G = H and H = {(s,0,0) | s e R}— the centre 
of G = H. It is the original inspiration for this approach I17U221|341 . 



IS 
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(ii) For the AHW group G and its representation J2.25|l with k = 1, take a 
function a[g,x) on X = T\G = G x G and calculate, cf. QZl (2.32)]: 



(4.3) 



[pCa)fl(g') = 



ff(g/X)x(g')f(g + g')dgdx 
6- 2 (g,g')f(g + g')dg 



G J 



G J 



&(£,,g'U(g)f(g + g')dgd£, 



ttg'W^g') 



f{g")Ug")dg"dE u 



here g" = g + g', &2 is a function on G x G, which is the Fourier transform 
of cr in second variable. The last expression l|4.3t coincides with Kohn- 
Nirenberg type PDO <T3J for a(g, £,) = &(£,, g), cf. (501 Part II]. 

(iii) For the Dynin group D, the unitary representation p d2.26l l on L 2 (H) 
is obtained from its infinitesimal action l|2.Hl and | |2.5l l. The integrated 
representation l|4.1ll was considered in [13 1 as a generalisation of the Weyl 
quantization from H to the group D. 

If function khas the structure k(z, t, u, v, s, x, y ) = 6(z,t,u,v)ki(s,x / y), 
where 5 is the Dirac delta function, then p(k) is a convolution on the 
Heisenberg group with the kernel ki . On the other hand, if k(z, t, u, v, s, x, y ) = 
6(z)k2(t,u,v)6(s,x,y), then p(k) is an operator of multiplication by £2 (s,x,y) — 
the (Euclidean) Fourier transform (t, u,v) — > (s, x,y) of k 2 . Thus, the inte- 
grated representation J4.1|l in this case belongs to the algebra of operators 
generated by convolutions on the Heisenberg group and multiplications 
by functions, which were investigated, for example, in 1 13, 14, 28, 43]. For 
a suitable choice of symbols, this operators coincide with J1.61 used in 1 3 ] . 
Furthermore, we can observe that in both cases kernels depend on the 
coordinate z through the delta function. Thus, instead of the integrated 
representation jA.l) we can use the relative convolutions d4.2|> for G = D 
and H being its centre, cf. the case of the Heisenberg group above. 

(iv) For G = SU(1, 1) and H = K, a substitution of j2.14|l into representa- 
tion l|2.28b produces the relative convolutions: 

dwA dw 



[p(k)v](z 



k(w) — v 

D wz + 1 



wz- 



(4.4) 



1-zC 
1-zC 



z-C, 

zC-1 



v(0- 



(1 - Iw|' 
dCAdC 

i-ici 2 ) 2 ' 



where w 



z-C 
zC-1' 



Interestingly, the last integral can also be interpreted as v(z) = (v, p (s _1 (z))k), 
which is the induced wavelet transform on K\SU(1, 1) ||39l § 5.5] with the 
mother wavelet k. 

The indicated connection of relative convolutions with the induced wavelet 
transform is not an exception. It occurs in many other cases when the represen- 
tation space V consists of functions defined on the homogeneous space X = H\G, 
e.g. the FSB space of analytic functions on C n = Z\H n . In general, the covariant 
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transform, the Berezin symbol, integrated representations and the contravariant 
symbol (considered below) are closely connected and, sometimes, even confused. 

4.2. Twisted Convolutions. It is desirable to have an efficient symbolic calculus 
of relative convolutions. For exponential Lie groups, a calculus in terms of the 
respective Lie algebras was initiated in [34]. However, the exponential property 
is rather restrictive, for example, SU(1, 1) does not posses it. Here we provide 
another algebraic condition, which is sufficient for relative convolutions to be 
closed under multiplication. 

In the notations of Section l2T2l for any x\, X2 G X = H\G there is the unique 
x G X defined by the identity 



(4.5) 



s(xi) s(x2) = hs(x), that is x = p(s(xi) s(x2)) = Xl • S(X2), 



where the last expression uses notation \2.7) . 

The relation (|4.5l l defines a binary operation {x\,X2) *-¥ x, which turns X into a 
semigroup. It is not a group unless H is a normal subgroup of G. One can develop 
a separate theory for semigroups from homogeneous spaces, for example, in [58J 
they are called gyrogroups. However, we prefer to proceed in terms of the original 
group G and its subgroup H. 

For given X2, x G X, there is the only Xi = X • (s(x2)) _1 G X satisfying the first 
identity in 04.5b . We will use the abbreviation Xi = xx^ 1 for it. 

Furthermore, using the transformation rule (12.161 of the measure dx on X we 
calculate: 



dxi dx2 



A H (h(x,x 2 )) 
A G (h(x,x 2 )) 



~-l/ 



~-l/ 



dx2 dx, where h(x,X2) = s(x)s (x2)s (x ■ s (X2)). 



Here, for simplicity, we write s _1 (y ) instead of the more correct expression (s(y ) ) _1 . 

Let two relative convolutions be defined by scalar-valued summable kernels 
ki, k2 G L^X). Then, starting with the Fubini theorem we calculate: 



p(ki)p(k 2 ) 



ki(xi)p(s(xi)) dxi 



k 2 (x 2 )p(s(X2))dX2 



ki(xi) k 2 (x 2 ) p(s(xi)s(x 2 )) dxi dx 2 



ki(xi)k 2 (x 2 ) p(s(xi)s(x 2 )) dxi dx 2 



-\i 



ki(xx 2 i )k 2 (x 2 ) p(h i (x,x 2 )s(x)) 



A H (H(x,x 2 )) 
A G (h(x,x 2 )) 



dx 2 dx 



i 1 -in 1 \ r-u-li ^ A H(h(x,x 2 

kj(xx 2 k 2 x 2 ) ph x,x 2 - r- dx 2 p s x dx 

x A G hx,x 2 



-i( 



(4.6) 

where 

(4.7) 



ki(xx 2 1 )k 2 (x 2 ) p(h (x,x 2 )) 



k(x) p(s(x)) dx, 



A H (h(x,x 2 )) 
A G (h(x,x 2 )) 



dx2 p(s(x)) dx 



k(x) = 



^ I -In I \ fu-lf u A H It x,x 2 . 

kixx 2 x k 2 x 2 ph x,x 2 )) TT dx 2- 

x A G hx,x 2 
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Note that, if the representation p is induced by a character x of the subgroup H, 
then p(h _1 (x,X2)) = x(h _1 (x,X2)). Thus, the above integral is scalar valued. 

Definition 4.2. For two summable functions ki and k2 on X = H\G, their twisted 
convolution k = kitfe is a function k, such that the relative convolution p(k) JA.21 
equal to the composition p(kj)p(k2) of relative convolutions with the kernels kj 
and k2, i.e.: 

P(kitjk 2 ) = p(k t )p(k 2 ). 

Then, the result of calculations (|4.61 l can be encapsulated in the statement: 

Proposition 4.3. The twisted convolutions k = k\ t[1c2 of two functions ki and k2 on 
H\G zs presented by l|4.7Jl . 

Remark 4.4. Integrated representations (|4.1) l and relative convolutions (|4.2) l map 
functions to operators. It is a fashion now to call any such map a "quantiza- 
tion". An opposite procedure, e.g. the covariant transform, maps an operator 
to a function — a symbol of the operator. This can be called "dequantization", 
respectively. Thus our Defn. 14.21 can be stated in quasi-quantum language as 
follows: quantize kernels to operators, compose operators, dequantize the com- 
position to the kernel. In this setup the twisted convolution is also known as a 
star product. We refer to [2J for further discussion, references and a more explicit 
formula in the case of square integrable representations. We also note, that a 
search of a compatible star product for an arbitrary Poisson manifold is the topic 
of deformation quantization. 

Example 4.5. (i) The Heisenberg group H and its centre Z are unimodular, 

thus Ah = 1 and Az = 1. Using maps p and s from Example |2.1(i) for 

M 2 = Z\H, we calculate: 

(x,u)(x 2 ,u 2 ) _1 =(x-x 2 ,u-u 2 ), 

h((x,u),(X2,U 2 ))=(i(X2U-UX 2 ),0,0). 

Thus, for a representations 112.211 and (|2.23l l induced by a character Xh(s, 0, 0) 
e 2mfls , the respective twisted convolution is: 



(k^k 2 )(x,u) 



kj(x - x 2 ,y - y 2 ) k 2 (x2,u 2 ) e-^^y-y^) dx2 dy2 



This operation is the key for the whole calculus of PDO as explained 
in ATI § 2.3; |22j § 2]. It is also known as the Groenewold-Moyal star 
product (2j § 6.1]. 
(ii) For an AHW group G, we calculate h((g,x), (g2,X2)) = X2(g2 — g). From 
unimodularity of G, the twisted convolution is: 



(Mk 2 )(g,x) 



ki(g - g2,x-X2) k 2 (g 2 ,X2)x2(g2 - g) dg 2 dx 2 - 



This again looks like a convolution on the Cartesian product G x G with 
a "twist". A composition with the Fourier transform on G x G maps our 
twisted convolution to the star product from [61 § 3]. 
(iii) According to Kirillov's theory, any unitary irreducible representation of a 
nilpotent group Lie group is induced by a character of the group's centre 
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C l25l § 15]. Thus, the relative convolution for C\D is not much different 
from the whole integrated representation. 

On the other hand, since the subgroup M JA.8} is normal then the 
twisted convolution for M\D = H reduces to the group convolution on 
the Heisenberg group. Some interesting options are located between the 
extremes C and M. For example, since the Lie algebra of D is generated 
by X, Y, T, it is worth to consider twisted convolution associated to the 
subgroup 

(4.8) M'={(z,0,u,v,0,0,0)eB I (z,u,v) e K 3 }. 

(iv) The group G = SU(1, 1) is not unimodular, however Ag(Jc) = 1 for all 
k in the maximal compact subgroup K [46, § III.l]. The subgroup K is 
unimodular since it is commutative. Furthermore, using maps l|2.14[l we 
calculate: 

-if i z ~ w u ^ |l-zw| 

z-s [w = -, h(z,wj = - — : — . 

1 — zw 1 — zw 

Thus, for the representation (|2.28|l induced from K, the respective twisted 

convolution is: 

1 — zw dw A dw 



(k^k 2 



kj k 2 (w 



D \l-zwj 1-zw n-lwl 2 ! 2 ' 



This corresponds to the composition of Berezin's contravariant symbols 
(see below) and nicely complements the well-known calculus of Berezin's 
covariant (Wick) symbols considered in Example |3.8(iv)| and J6j § 4.2; [HJ 
|59] § A.3]. 

4.3. Contravariant Symbol and Toeplitz Operators. There is a notion immedi- 
ately derived from the integrated representations: the contravariant (aka inverse 
covariant) transform 1 37-39. 44]. For an integrated representation p l l4.lt (or l|4.2t ) 
and a fixed vector w e V the associated contravariant transform of a function k 
on G (or X = H\G) is 

(4.9) M£,(k) = p(ic)w, where Jctg^kCg" 1 ). 

The contravariant transform M^, intertwines the right regular representation R 
on L 2 (G) and p: 

(4.10) MP,R(g) = p(g)JvC. 

Combining with d3.3b , we see that the composition Mw o Wj of the covariant and 
contravariant transform intertwines p with itself. We can use the Schur's lemma 
Lem. 4.3.1;H3 Thm. 8.2.1] to deduce that: 

Proposition 4.6. For an irreducible p, the composition M„ o Wj is a multiple 0(w,v)I 
of the identity operator. Moreover, the factor 8(w,v) is a sesquilinear form of vectors w, 

veV. 

The following interesting consequence is known in slightly different form for 
the case of the Heisenberg group [17. (1.47)]. 

Corollary 4.7. Assume that the integrated representation p is faithful on the image space 
W V2 V. Then the twisted convolution of wavelet transforms is: 

(4.11) W Vl u^W V2 u 2 = e(u 2 ,v 1 )W V2 u 1 



22 VLADIMIR V. KISIL 

Proof. We note another form p(W v u)w = 9(w,v)u of the identity l[ o W? = 
0(w,v)I. Then: 

M V2 (W V] u^W V2 u 2 ) = p(W Vl u^W V2 u 2 )v 2 

= p(W Vl U!)p(W V2 U2)V2 

= p(W Vl u 1 )0(v 2/ v 2 )u 2 

= 0(v 2/ v 2 )p(W Vl U!)u 2 

= 0(v 2/ v 2 )9(u 2/ vi)ui. 

Since the representation p is faithful on the image space W V2 V, the obtained result 
implies ( |4.1H . D 

The following particular case of (|4.11b is of special interest: 

Corollary 4.8. Under the assumptions of the previous Corollary, the image space W f V 
is reproducing kernel space with the following realisation of a reproducing formula: 

(4.12) v = ftiv, for f = W f f and any w G W f V. 

The contravariant transform is a source of the Berezin's contravariant symbol 
as follows. For a pair v G V, f G V*, consider a rank-one operator E V/ f : V — > V 
define by the expression E V; fU = (u, f ) v. Then, the representation p l!3.13l l acts as 
follows: 

(4.13) p(gi,g 2 )E v ,f = E vV f, where v' = p(g 2 ) _1 v and f' = p*(gi)f, 

with the last identity natural meaning (u,f ) = (u,p*(gi)f) = (p(gi)u,f). Then, 
the contravariant transform for the vector w = E v ,f G B (V) becomes: 

(4.14) Mv, f (o) = 
Using J4.13L we can directly write the action of operator J4.14II on u G V: 



x 



a(xj,x 2 ) p(s(xi),s(x 2 ))E V/f dxi dx 2 . 
x 



a(xi,x 2 ) (p(s(xi))u,f) p(s(x 2 ) 1 )vdxidx 2 . 
x 



(4.15) Mv,f(a)u = 

Jx . 

Here we call a the contravariant symbol of the operator M v ,f (a). 

The contravariant symbol in the sense of Berezin [4-6 1 appears if V is a Hilbert 
space with an irreducible square integrable representation p. The covariant trans- 
form W v for an admissible mother wavelet v G V identifies V with its image W V V. 
There is the respective reproducing kernel k y | |3.6l l on W V V. If the representation 
p is restricted to the diagonal (3(g) = p(g, g) of G x G, then the contravariant 
transform similar to 04.151 1 is [5, § 1.1]: 



[M v ,v(a)u](y) = 



(4.16) 



a(x) (p(s(x))u,v) p(s(x) 1 )v(u)dx 
x 



a(x)u(x) k y (x) dx. 



The last expression is the Toeplitz operator T a = PaP for the projection P on W v 
defined by the integral in the right-hand side of the reproducing formula (|3.6b . 

The explicit formulae connectioning co- and contravariant symbols are known 
for a long time [5 (1.12); 6 (3.13)]. Within our approach they are consequences of 
Prop. |4~6l since covariant and contravariant symbols are special cases of covariant 
and contravariant transforms. 
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The original Berezin's papers [4-6] (as well as subsequent developments in the 
context of abstract reproducing kernel spaces) do not assume any group struc- 
ture. It is possible to obtain important estimations for norm and compactness 
in this abstract setting. However, the fundamental examples — the Bergman and 
FSB spaces — considered in those papers are generated by groups, as we have 
seen above. In particular, the group structure becomes very relevant in the study 
composition operators generated by an automorphism of the domain [ 8 1. Further- 
more, the formula for twisted convolution d4.7b is also based on the underlined 
group structure and is not possible on a generic reproducing kernel space. 

Example 4.9. (i) For Toeplitz operators (|3.18b on the Heisenberg group, the 

contravariant calculus was already investigated in [4] and is still an im- 
portant tool OH). The connections between PDO dl.lb and the Toeplitz 
operators (|3.18b was fruitfully exploited in [22, § 4.2] with the following 
observation: "The Toeplitz operators are to the Bargmann-Fock model 
as the pseudodifferential operators are to the Schrodinger model". We 
used such a technique to obtain Calderon-Vaillancourt-type estimations 
for relative convolutions on exponential nilpotent Lie groups |45|. 
(ii) For the AHW group G, the connection between PDO-type operators l|4.3t [50 
and Toeplitz-type operators on L 2 ( G x G ) shall closely follow the Heisen- 
berg group suit. Yet, no work in this direction is known to me. 
(iii) For the Dynin group, I am not aware of any study of contravariant cal- 
culus and Toeplitz-type operators. However, it is natural to expect, that 
their relation to the PDO-like calculus from Example |4.1(iii) and [3 13-15] 



shall be similar to the Heisenberg group. However, due to a higher level 
of non-commutativity it may be not as straightforward as for an AHW 
group, 
(iv) For SU(1, 1) and Toeplitz operators l|3.20l l, the Berezin contravariant sym- 
bols was studied in (6L with numerous fruitful developments, cf. llMTOl 

CLUED. 

5. Discussion 

The moral of the present overview is that there is no a single formula perfectly 
serving all situations. However, covariant and contravariant transforms provide 
a general framework which has a rich and flexible inventory. The presented list 
of different examples prompts further detailed investigation of this approach in 
various directions. 

Appendix A. Covariant Transform: A Road Map 

This paper's presentation was illustrated by numerous detailed examples. Here 
we provide a brief outline of notions and main formulae used in our construc- 
tions. 



duced representations. 

- G is a locally compact group with a right invariant measure dg and 

the modular function Aq- 
-He G is a subgroup with a right invariant measure dh and the 

modular function Ah • 
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- X = H\G is the homogeneous space of right cosets: gi ~ gi if gi = 
hg2 for h G H. 

- p : G — > H\G is the natural projection of an element to its coset. 

- s : H\G — > H is a section — a right inverse of p: p(sx) = x for all x G 
H\G. The map s is not unique and we often can chose it continuous. 

- r : G — > H is defined from the identity: g = r(g)s(p(g)), g <G G. 

- X = H\G is a right G-space with the action: g : x i->- x • g = p(s(x) * g), 
g e G, x g X. 

- A representation of G induced by a character x of the subgroup H is 
[p x (9)flM=x(r(s(x)*g))f(x-g). 

- An equivalent form of the induced representation is by the right 
shift R(g) : F(g') i->- Y(g'g) on a space of functions with the property 
F(hg) = x(h)F(g), for h G H and g G G. 

- Two models of induced representations are connected by lifting [£ x f] (g) 
X(h)f(p(g)) and pulling [?F](x) = F(s(x)). 

• Covariant transform. 

- For a representation p of G in V and an operator F : V — > U, the 
covariant transform is [Wpv](g) = F(p(g)v), where v G V and g G G. 

- The induced covariant transform is [Wpv](x) = F(p(s(x))v) for v G V, 
x G H\G. 

- If a mother wavelet f satisfies to the identity J G a(g) p*(g)f dg = 
for some distribution a(g) on G, then any wavelet transforms V = 
(v, p*(g)f) satisfies the identity Dv = 0, where D = J G a(g) A(g) dg 
for the left regular representation A. This conditions can characterise 
the image WfV among all functions on G or X. 

- Often, the image of (induced) covariant transform has the repro- 
ducing property f(y) = J x f(x) ky(x) dx, where k y is the covariant 
transform of the shifted mother wavelet p(s(y) _1 )v. 

- For a representation p of G in a vector space V there is a represen- 
tation p(gi,g 2 ) : A <-¥ p(g 2 ) _1 Ap(gi) of G x G on the space B(V) of 
bounded linear operators on V. 

- The covariant symbol A(gi,g2) = F(p(gi,g2)A) = F(p(g2) _1 Ap(gi)) 
is the covariant transform defined by the representation p and the 
operator F : B(V) -> U. 

• Contravariant transform. 

- For a representation p of G and a summable function k on G, the 
integrated representation is p(k) = J G k(g) p(g) dg. 

- The relative convolution is an integrated representation over a ho- 
mogeneous space p(k) = J x k(x) p(s(x)) dx. 

- The composition of two p(ki)p(k2) relative convolutions produces 
the twisted convolution pfkitfe) = p(ki)p(k2). 

- For an integrated representation or relative convolution p and a 
fixed vector w G V the contravariant transform of a function k is 
OVC(k) = p(k)w, where k(g) = kfg- 1 ). 

- The twisted convolution of two wavelet transforms is W Vl uit]W V2 U2 = 
0(u 2/ vi)W V2 ui. 
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